NCEI
NOAA's National Centers for Environmental Information (NCEI) hosts and provides public access to one of the most significant archives for environmental data on Earth. Through the Center for Weather and Climate and the Center for Coasts, Oceans, and Geophysics, they provide over 25 petabytes of comprehensive atmospheric, coastal, oceanic, and geophysical data.
The interface located on via the NCEI page, found under the Data Management menu item, is intended for data submission to these national archives for SCS users on one of OMAO's large vessels. If you are not on one of these vessels please see your data coordinator for instructions on how to submit your SCS data (R2R, S2N, etc) to NCEI.
More details regarding your NOAA ship can be found on the SDALs Data Management page - https://sdat.noaa.gov/DataManagement/Tracking
Meta Data Management
The meta data provided to NCEI is pulled from the sensor configuration defined by CFE and the general system meta data you have described throughout SCS (such as your vessel profile page). You cannot directly send meta data to NCEI, however it is automatically pulled together and sent for each day that is submitted to NCEI, be it current or historic.
Meta-data defined outside CFE does not maintain a history, if you submit historic data the configuration of your sensor suite at that time will be sent, however your current ship metadata values (not those active at that time) will also be submitted. This may result in a mis-match should you wait to long to submit your datasets!
Submission Management
Submissions to NCEI can be reviewed and controlled via the NCEI page under the Data Management menu item.
The UI is broken down into two columns. On the left you will find your controls, a summary of your current status and the list of problematic prior submissions. On the right you will find a grid displaying all recent and active/current submission jobs.
Auto Submission
At the top of the control column you will find a checkbox which tells SCS to automatically submit the days data to NCEI every night. If this is checked then at midnight SCS will gather all sensor data collected (by SCS) for the prior 24 hours, package it up along with all relevant metadata and push it off to NCEI. If you know bandwidth will be limited, or you want to assume manual control, simply uncheck this box and SCS will no longer automatically submit data on your behalf. When you check the box again the auto submission will resume, however if you miss any submission windows you will have to manually send those days in as the software does not look back and fill in the gaps!
Manual Submission
To manually submit data all you have to do is enter a Start date and optionally an End date. Then click the Submit Date Range button.
If you do not enter an end date then all days from the start date to 'now' will be [re]submitted.
If you enter the same date (start date equals end date) then only that single day will be submitted.
Once you click the Submit Date Range button you should see the job appear in the submission grid located on the right.
Submission Summary
Below the auto and manual submission controls you will find a grid and graph depicting your ships current data submission status. Based upon your schedule in SDAT NCEI will expect a certain number of days worth of data from you. Obviously this number is only so accurate, deviations due to schedule changes and a multitude of other reasons will screw the number in one direction or the other. In those cases NCEI might reach out to you to [human to human] figure out what's going on and correct their expectations on their end.
Generally speaking however, in this section you can see a breakdown on how well your submitting your data. The prior year will be broken down into the categories defined below.
- Missing
- Days the NCEI expected data which have not yet been sent in
- Received
- Days sent in and acknowledged as having been received by NCEI, pending processing
- Corrupted
- Days which made it into NCEI but were unable to be published due to errors
- Published
- Days which have been officially processed and made available to the public.
In an ideal world all expected days will be Published, but in reality submissions might get corrupted in transit, days might get missed or deleted, etc. You can work with the OMAO data manager and/or NCEI themselves to resolve these issues as they occur.
Resubmission of Data
Below the submission summary you will find two lists. Inside these lists are the days which NCEI believe to be missing and the days which you submitted but NCEI found to be corrupted.
SCS will not automatically resubmit any of your data for you. Any days deemed to be missing (SCS was off, ships schedule changed, Auto-Submission was disabled, etc) will have to be resolved manually. Any corrupt days (zipping up the submission package didn't "zip" right, packets didn't transmit over the VSAT correctly, etc) will also have to be manually resolved. These flags are set on the shore, the only way to get them off the list is to have someone on shore clear them. This can be done by a variety of people, the primary POCs being the OMAO Data Manager or NCEI. Potentially Chief OPS at either MOC could also deal with it if schedule related.
The first step however is to simply submit or resubmit the days. This is easily accomplished by clicking the Submit Day button (or the Submit All button found at the bottom of the list). If a day is resubmitted and continues to present issues then at that point you would want to get someone on shore involved.
Once a day makes it to Published your responsibility for managing it has ended. That days data is now fully backed up and available on shore, you can and should delete them from your ship as they can be retrieved from the National Archives as needed.
Recent and Active Submissions
On the right hand side of the page you will find a grid listing all recent and currently in-progress submission jobs for NCEI. Each job is a days worth of data and can be in one of 4 states.
A spinning cog icon will tell you which state each job (a row / date) is in. When complete it will change to a green checkbox and move to the next step.
The first step is extraction of the datasets from the database. This is essentially a data dump for the entire day and could take some time depending on how large a sensor suite you have and how much data you recorded for that day.
After the data is extracted it is packaged (generally into a NetCDF format along with various XML encoded files) in a manner NCEI understands and can ingest, it is then zipped up in preparation for transmission.
When the package is built it can be sent to NCEI a variety of ways. This could be very fast if simply transferred locally for a ship-to-shore sync (Aspera, rSync, NetMan, etc) or sent over a high speed connection. It could also take quite some time if actually sent to shore over the VSAT or other low speed connection.
As of the time of this manual being written there is no way for you, as a general user, to decide which mechanism is used to send the data to NCEI. It is still pending discussions between OMAO, MO, the other line offices and NCEI.
Once all columns are marked in green then the job is completed and you can look forward to the day no longer being marked as Missing by NCEI.
If a job gets screwy or you want to cancel it for whatever reason you can always abort it via the button in the last column.